1. Field of the Invention
The present invention relates generally to control of exposure times of color digital cameras.
2. Discussion of the Related Art
The old adage that xe2x80x9ca picture is worth a thousand wordsxe2x80x9d has taken on a heightened degree of meaning in the wake of the development of digital image processing technologies. Digital image processing methods enable pictorial data to be manipulated in ways that enhance the means by which humans can interpret this data. Initially, this capability gave rise to an interest in transforming images, stored via traditional media products, into digital formats. Typically, this has been performed by use of an image scanner. More recently, efforts have focused on methods of directly creating digital images. Digital cameras are an end product of these efforts.
Unlike traditional film cameras, digital cameras use a solid-state device called an image sensor to capture the image. Image sensors are semiconductor chips containing a grid of hundreds of thousands to millions of photosensitive diodes called photosites. Each photosite corresponds to a single pixel of the captured image. Light energy entering the digital camera impinges on the image sensor causing the semiconductor material to release electrons that are captured by the photosites. In this way, each photosite records, as an electric charge, the accumulation of light energy that impinges on it. As a photosite is exposed to light for longer periods, increasing amounts of light energy reach it, and a greater electric charge is stored. The closing of the shutter initiates a process whereby the charge at each photosite is measured and converted to a digital representation. All photosites, taken in aggregate, store a digital mapping of the image that can be used to reproduce it.
While recording the amount of light energy that reaches them, photosites do not distinguish this light energy by its different wavelengths. Therefore, photosites are, by themselves, insensitive to color. Digital cameras perceive color by use of color filters placed between the impinging light and the photosites. A color filter passes light energy at or near the wavelength corresponding to the color of the filter and attenuates light energy outside this band. This has the relative effect of favoring the accumulation, on a photosite, of light energy at wavelengths at or near the wavelength corresponding to the color of the filter. This allows digital cameras to leverage a natural phenomenon of visible light. All of the colors of visible light can be realized through various combinations of the three primary colors of red, green, and blue. Therefore, it is possible, by using filters set to these three primary colors, to capture the various hues of an image and reproduce the actual coloring of the scene by reassembling the various filtered components.
Color filtering typically occurs by one of two methods. One process requires taking a series of exposures in which a different colored filter is used with each iteration. These filters can be individual color filters or a composite unit that can be controlled to filter different colors. Composite filters using liquid crystal display technology are commonly used. Light passing through a given filter reaches each of the photosites on the image sensor. This facilitates a high degree of accuracy when the image is reproduced. However, this high degree of accuracy depends on the ability to hold the objects in the image stationary for the time necessary to record the three images and change the filters between each exposure. A modification of this approach involves using a digital camera with three image sensors. A different colored filter is used for each sensor so that all three primary colors of the image are captured in a single exposure. Digital cameras with three image sensors are relatively expensive.
Another alternative method involves affixing an individual filter to each photosite such that the overall image sensor appears as a mosaic. The Bayer pattern is a typical arrangement of photosite filters. Empirically derived to match the perceptive qualities of the human eye, it uses twice as many green filters as blue or red. Using filters affixed to and covering individual photosites allows all colors of an image to be captured in a single exposure. True colors are divined from the three primary colors by use of a comparison algorithm. The degree of accumulated energy recorded at a given photosite is compared with the levels stored at surrounding photosites to determine the true color of the image at the position corresponding to the given photosite. This is the color that is rendered when the image is reproduced. Owing to the large number of photosites, performance of the algorithm requires a considerable amount of processing time.
While this approach allows all colors of an image to be captured in a single exposure, the fidelity of the reproduction suffers from the fact that each primary color is recorded by only a portion of the total number of photosites. Accuracy is also impeded by reliance on the interpolation algorithm to estimate the true color of a given location on the image. Additionally, a certain amount of the accumulated electric charge at a given photosite results from electrons released from the semiconductor material due to thermal energy. The interpolation algorithm can be limited in its ability to correct for variations, among the photosites, in this thermal noise.
Image sensors traditionally have used charge-coupled devices (CCDs) for their photosites. Development of this technology has enabled the production of high quality image sensors. More recently however, image sensors have been made using complementary metal oxide semiconductor (CMOS) technology. With CCD technology, after an image is captured, stored charges are transferred from the photosites row by row to a readout register for subsequent processing by devices on other chips. As charges stored in the row nearest the readout register are transferred to it, charges stored in the next adjacent row are transferred to the row nearest the readout register. Charges stored in all other rows are likewise transferred to their respective adjacent rows nearest the readout register. The charges on each row are xe2x80x9ccoupledxe2x80x9d to those on the adjacent row so that when one row moves, the next also moves to fill the vacated photosites.
As a category of electron devices, CCDs have limited applications. They are fabricated on wafers using relatively expensive processes at foundries that specialize in their production. In contrast, CMOS technology is widely used for chips supporting a variety of electronic products. Therefore, use of CMOS allows manufacturers to enjoy tremendous economies of scale and to reduce their manufacturing expenses substantially. Additionally, development of CMOS processing technologies has realized the manufacture of wafers with high yields of useable chips, further reducing costs. Also, image sensors made from CMOS can include processing circuits on the same chip. Yet, while CMOS technology offers several advantages over CCDs, fabrication of image sensors from CMOS is still in its early stages of development and CMOS imagers do not yet have the quality of CCD sensors.
Digital cameras offer several advantages over other imaging technologies. With a digital camera, one can immediately view the captured image on a screen, transfer the image in a digital format via a host of communications technologies, display the image via a variety of media, edit images with digital processing techniques, save the costs of buying and developing film, reduce the use of toxic chemicals used in traditional photography, and ergonomically configure the design of the camera for use with other equipment, such as a microscope. With digital image processing, images can be cropped to emphasize specific portions, expanded or reduced in size, altered for brightness, blended with other images, or filtered to sharpen or blur outlines of objects or create a special effect. While it is possible to extract a still image from a videotape using a frame grabber, images created with digital cameras have higher degrees of resolution and are less likely to deteriorate during duplication or due to age. Finally, because digital imaging is not constrained to standard aspect ratios, digital cameras, outfitted with the right interface boards, are readily adapted for use in systems where they can be configured to match the optical characteristics of other pieces of equipment.
Because of these advantages, digital cameras have been sought to be used to document scientific work products in such fields as biochemistry, cell biology, pathology, genetics, forensics, geology, metallurgy, inspections of electron devices, quality control applications, and the like. The high degree of resolution and ease of configuration have made digital cameras a particularly desirable tool for capturing microscopic images. Early uses in this field revealed difficulties in coupling digital cameras to microscopes or endoscopes. These difficulties have stemmed not so much from the mechanical coupling of the devices as from the complications that arise when lenses, cables, and interface boards are added to the process.
Fortunately, interest in this use has been sufficiently large enough to warrant industry response to these problems. Manufacturers of digital cameras have been quick to develop turnkey systems that combine a digital camera, cabling, and interface boards, all designed for use with a microscope. The cameras themselves often integrate an image sensor, an analog-to-digital converter, and a processor into a single unit. Some packages even include software to aid the user in configuring the parameters, operating the camera, and processing the images. Software tailor-made by the manufacturer can exploit the strengths of the digital camera system while diminishing its weaknesses. The software can also automate several basic functions so that end users need not familiarize themselves with unnecessary aspects of the functioning of the system. Furthermore, software can be used to customize operations of the system for specific applications.
The ability to capture microscopic images in digital form is especially useful for realizing the full potential of array technology. By supporting simultaneous examination of up to thousands of samples, each sample having a diameter on an order of magnitude of 200 microns, array technology has dramatically expanded the ability of researchers to perform a variety of biotechnological operations. Measurement and analysis of signals emitted from an array of objects usually involves examination of an image of the array. Typically, samples are labeled by use of fluorescent dyes or radiolabels. The labeled samples are excited and a detector system captures an image of the emitted energy. Accuracy of subsequent signal analysis is heavily dependent on the parameters of the detector system and its ability to reproduce faithfully the image of the array. Signals can be measured for absolute intensities or, in the case where different colored dyes are used to detect the degree of presence of different compounds, for ratios of intensities within specific frequencies.
The use of different colored fluorescent dyes and the importance of accurately measuring their intensities are characteristics that lend themselves to the methods by which digital cameras can be made to distinguish different colors. To date, the bulk of developmental activity regarding the display of colors on digitally reproduced images has focused on accurately integrating individual exposures filtered to highlight the primary colors of red, green, and blue. What has been needed in the art is a means to control, selectively, the highlighting of these colors on a digital image so that users can better extract the signals conveyed by the intensities or patterns of a given color and better attenuate the noise embodied as other colors.
The present invention provides a system and method for establishing an aggregate degree of brightness for each primary color to create a composite color digital image. The present invention is designed to function with different types of color digital cameras. Color digital cameras can be designed to use one or more image sensors and a variety of filtering mechanisms and configurations. Furthermore, image sensors can be fabricated using different technologies. To create a composite color digital image in which the aggregate degree of brightness for each primary color is set to a desired level; the present invention processes each color individually. Therefore, the present invention takes a series of exposures in order to capture all of the colors.
One of a plurality of primary colors is selected. A color digital image is captured with a color digital camera. The aggregate degree of brightness for the selected primary color is determined. In a preferred embodiment, the aggregate degree of brightness is determined by preparing a histogram of the values stored in those photosites corresponding to the selected primary color. Where filtering is accomplished by use of individual filters affixed to and covering each photosite on the image sensor, those photosites corresponding to the selected primary color are identified by the pattern of the filter. In this case, the histogram is prepared using only the values stored in the identified photosites. Otherwise, where filtering is accomplished by use of filters that cover the entire image sensor, all photosites are used to capture each color. In this case, the histogram is prepared using the values stored in all the photosites.
The determined aggregate degree of brightness for the selected primary color is compared with the desired degree of brightness for the selected primary color. If the determined aggregate degree of brightness for the selected primary color is not within a tolerance band surrounding the desired degree of brightness for the selected primary color, then an adjustment is made to the exposure time controller of the color digital camera. If the determined aggregate degree of brightness for the selected primary color is less than the lower value of the tolerance band surrounding the desired aggregate degree of brightness for the selected primary color, then the exposure time controller of the color digital camera is set so as to increase the exposure time of the color digital camera. If the determined aggregate degree of brightness for the selected primary color is greater than the upper value of the tolerance band surrounding the desired aggregate degree of brightness for the selected primary color, then the exposure time controller of the color digital camera is set so as to decrease the exposure time of the color digital camera. The process of capturing a color digital image, determining its aggregate degree of brightness for the selected primary color, comparing the determined aggregate degree of brightness with a desired aggregate degree of brightness, and adjusting the exposure time controller is repeated until the determined aggregate degree of brightness for the selected primary color is within the tolerance band surrounding the desired degree of brightness for the selected primary color.
If the determined aggregate degree of brightness for the selected primary color is within the tolerance band surrounding the desired degree of brightness for the selected primary color, then photosite values for the selected primary color are copied into corresponding pixels in a buffer. The process is repeated for each remaining primary color, thereby creating a composite color digital image in which the aggregate degree of brightness for each primary color is set to a desired level.